A number of systems include and employ depth sensing devices, such as time-of-flight (ToF) cameras, for tracking hands (for hand gesture recognition and input) or other objects. Such tracking is useful in applications with real-time processing demands such as robotics, vehicle guidance, and providing low latency gesture-based input, and for virtual reality (VR), augmented reality (AR), and mixed reality (MR) applications where low latency depth computation is desirable and yields improved performance and/or an improved user experience (for example, by reducing latency between user motion and displaying graphical elements to a user that correspond to tracked objects). For many such applications, depth sensing devices that cover a wide field of view (for example, greater than 120 degrees), offer high angular resolution (permitting more detailed modeling of a tracked object), and also operate with a high frame rate (for example, 60 frames per second or more) are preferred. However, ever-increasing resolutions and frame rates for depth sensing devices comes at a cost of increased amounts of camera data and shortened real-time deadlines for processing each frame of camera data.